On the Separability of Attractors in Grandmother Dynamic Systems with Structured 

Connectivity 



Luciano da Fontoura CostcQ 

Instituto de Fisica de Sao Carlos, Universidade de Sao Paulo, Av. Trabalhador Sao Carlense 400, 
Caixa Postal 369, CEP 13560-970, S&o Carlos, Sdo Paulo, Brazil 
(Dated: 22nd Dec 2006) 

The combination of complex networks and dynamic systems research is poised to yield some 
of the most interesting theoretic and applied scientific results along the forthcoming decades. The 
present work addresses a particularly important related aspect, namely the quantification of how well 
separated can the attractors be in dynamic systems underlined by four types of complex networks 
(Erdos-Renyi, Barabasi- Albert, Watts-Strogatz and as well as a geographic model). Attention is 
focused on grandmother dynamic systems, where each state variable (associated to each node) 
is used to represent a specific prototype pattern (attractor). By assuming that the attractors 
spread their influence among its neighboring nodes through a diffusive process, it is possible to 
overlook the specific details of specific dynamics and focus attention on the separability among 
such attractors. This property is defined in terms of two separation indices (one individual to each 
prototype and the other considering also the immediate neighborhood) reflecting the balance and 
proximity to attractors revealed by the activation of the network after a diffusive process. The 
separation index considering also the neighborhood was found to be much more informative, while 
the best separability was observed for the Watts-Strogatz and the geographic models. The effects of 
the involved parameters on the separability were investigated by correlation and path analyses. The 
obtained results suggest the special importance of some measurements in underlying the relationship 
between topology and dynamics. 



'Nothing in excess.' (Delphic proverb) 



I. INTRODUCTION 



Most dynamic systems, from neuronal networks to pat- 
tern formation, involves several interconnected variables, 
which can be properly represented in terms of a graph 
(e.g. [Hi) or complex network (e.g. 0, S, 1, 1 B S 1). 
Such systems are henceforth called complex dynamic sys- 
tems. A state variable v(i) is normally associated to each 
node i, so that the complete evolution of the system can 
be described by the N X 1 vector v^K In a neuronal 
network, for instance, each neuron can be expressed as 
a vertex (or a node), while synapses are represented by 
links (or edges) and the node activation by the respec- 
tive state variable. Once the underlying connectivity of 
such a syste m is represented in terms of its weight ma- 
trix W [lOOj , several important types of dynamics can be 
subsumed as 



lit+i) 



f(Wv 



(1) 



Any layer of the perceptron neuronal network (e.g. 
[lOl HU), f° r instance, is obtained by substituting /() 
by some abrupt function (e.g. hard limit or sigmoid). 
At the same time, the complete classic Hopficld model 
(e.g. [ill, [13]) can be represented by this equation. A 
simpler, linear model is obtained by making f(x) — x, so 
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that^* +1 ) = Wv^. In case W is also a stochastic matrix 
(the transition matrix), this linear equation subsumes all 
first order Markov chains, a particularly useful and im- 
portant dynamic model which is intrinsically associated 
to random walks ( e.g. (HI), Markov chains (e.g. [HI) and 
diffusion (e.g. [IE [ll). 

In many dynamic systems (e.g. [HI, 113 )> the codi- 
fication of external stimuli as well as the results from 
the network dynamics take place in the N— dimensional 
space defined by the state variables. For instance, one 
particularly pattern may be represented as the state 
v\ — [1, 0, —1, 0.5, 1, 0], while another pattern may be as- 
sociated to V2 = [0, 1, —1, 0, 1, 0]. This type of represen- 
tation is called vector coding (e.g. [l8lll~9|). States which 
are near any of such pattern-states are frequently under- 
stood to be associated to that pattern. For instance, in 
the case of the example above, a third pattern similar to 
v{ will tend to be coded as a state vector such that 
5 = \\v\ — U3II is small. Such a smooth, graded coding 
along the state space is interesting because it allows for 
some robustness/redundancy and flexibility/generality 
for the representation of the patterns, while also favor- 
ing good dynamic properties such as where the patterns 
are accessed through gradient-descent or similar methods 
(e.g. PI). 

However, it is also possible and interesting to consider 
the situation where the patterns are associated to spe- 
cific nodes, not to specific points in the state space, such 
as in systems involving the so-called grandmother cells 
(e.g. [13, HH H3] ) • Extremely important systems includ- 
ing great part of the mammals cortex are believed to be 
so organized. In these cases, a pattern is associated to a 
node i such that high values of v(i) signalizes the pres- 
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ence of that pattern. This type of cells is ubiquitous in 
several cortical areas (e.g. [2J, [24], [HI). For instance, 
cells which are highly specific to hands and faces have 
been found in the inferior temporal cortex [2l|, [2(| [27| • 
It is also known (e.g. [H [H, S HI ) that much of the 
mammals cortical architecture is characterized by spatial 
smoothness, in the sense that cells which are spatially 
close tend to exhibit similar dynamics and response (i.e. 
state correlations), except at eventual singularities in- 
volving fractures |25l . [2^ | . Such an organization reminds 
of the smooth coding discussed above for dynamic sys- 
tems where patterns are represented by the overal state 
(i.e. vector coding). It is important to note that the 
smoothness propert y is not exclusive to topographically 
organized systems [10l| such as the mammals cortex. 
In other words, even in networks where the nodes have 
no specific position in an embedding space, neighboring 
grandmother nodes may tend to have similar dynam- 
ics and response. In such cases, it is interesting to use 
the concept of progressive neighborhoods around a node 
(e.g. 0, M, Hi Hi, H i|)), defined by the immediate 
neighbors of a node as well as its second neighbors, and 
so on. Following [H, [H, HH, in this work we organize 
the successive neighborhoods in terms of their hierarchi- 
cal level. Therefore, the immediate neighbors of a node 
i are at hierarchy 1 of i, and so on. In brief, the coding 
smoothness property in grandmother systems is reflected 
by the fact that the hierarchical neighbors of each node 
will tend to have responses similar, though progressively 
diverging, to that of node i. 

In dynamic networks with a finite number of nodes, 
the situation considered herein, a particularly interest- 
ing problem arises as a consequence of a tension between 
the grandmother coding and the smoothness property, 
a phenomenon directly related to the limited number of 
patterns which can be properly represented (e.g. [III). 
Before proceeding further, it is important to provide a 
more objective characterization of such a problem. Con- 
sider that M prototypic, distinct, patterns are to be rep- 
resented in the network. If each of such patterns is rep- 
resented by a respective prototypic neuron (node) , in the 
sense that this neuron will be the most highly activate 
when that pattern is invoked by the network, we also 
need to allow for intermediate cells with graded replies 
between pairs of nodes. Observe that such prototype 
nodes typically act as attractors for the dynamics of the 
respective dynamic system, being accessed, for instance, 
by using gradient descent and/or random walks methods. 
If M is too large, a point will be reached where each 
network node will be required to represent each proto- 
typical pattern, running out of nodes for implementing 
the graded responses. Although this situation occurs at 
the very limit of the network capacity, it is important to 
observe that it may still provide a complete, invertible 
representation of the patterns (i.e. an one-to-one map- 
ping). On the other hand, the smoothness of the coding 
is undermined as the number of prototypical patterns 
increases. In addition to destroying the network poten- 



tial for generality and robustness, it will also become im- 
possible to retrieve the patterns by gradient-descent like 
mechanisms along the network. Figure Q] illustrates two 
small topographic state spaces: one (a) corresponding 
to the limit situation where each pattern is associated 
to each node, the other (b) depicting a smooth map de- 
fined by diffusion around five prototype nodes. These 
two spaces were assumed to have local connectivity, in 
the sense that each cell communicates only with its most 
immediate neighbors in the orthogonal lattice. Observe 
that it is virtually impossible to devise an effective re- 
trieval or activation mechanism capable of getting to any 
specific prototype node in Figure HJa) . In addition, the 
failure of any cell will imply in the irrecoverable lost of 
one pattern (the origin of the gradmother cell concept). 
On the other hand, the state space in Figure [IJb) al- 
lows a good level of tolerance to failure (this also means 
some redundancy), at the same time as simple dynamic 
mechanisms such as gradient descent can be employed 
for retrieving and activating nodes. 




(b) 

FIG. 1: Gray-level visualizations of a topographic state space 
with too many prototypical patterns (a) and of a smooth to- 
pographic state space (b). Both these spaces are underlined 
by 4-neighborhood in the orthogonal lattice. The five proto- 
type patterns in (b) have been marked in black for the sake 
of better visualization. 



For all that has been said above, it becomes very im- 
portant to quantify and study the pattern separability in 
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dynamic systems such as neuronal networks. In the case 
of the dynamic systems which can be fully represented 
by Equation [U these two properties are necessarily im- 
plied by the network connectivity, defined by the weight 
matrix W, and the function /(). The present work fo- 
cuses precisely on the investigation of the separability 
between patterns in grandmother complex dynamic sys- 
tems. By considering a diffusive process emanating from 
each prototype node, followed by a process which assigns 
to each edge the difference between the activation states 
at its head and tail nodes, a graded representation of 
the prototype patterns can be obtained. This allows full 
generality of the reported investigations as an approxi- 
mation to many types of grandmother dynamic systems. 
The accessibility to the original prototypes is simulated 
through a preferential random walk (diffusion) starting 
at every node of the network. After a given relaxation 
time, the activation of the network is normalized so as to 
correspond to occupancy probabilities and compared to 
the original prototypes position. This is done by consid- 
ering two separation indices, one individual, considering 
the resulting activity only at the prototype nodes, and 
the other considering the activation not only at these 
nodes, but also at their immediate neighbors. These two 
indices, are henceforth abbreviated as Si n d and s, are 
obtained in terms of the normalized geometric average 
of the activations at (or around) each prototype node. 
Therefore, the maximum value of these indices (equal 
to one) is achieved only when the resulting activation is 
totally and uniformly distributed among only the pro- 
totype nodes (for Si n d) or among these nodes and their 
immediate neighbros (for s). 

The performed simulations and analyses first consider 
specific network configurations, in order to gather in- 
sights about the influence of each involved parameter, 
which are identified and discussed. Subsequently, a more 
systematic investigation of some parameter variations are 
performed. In order to infer the influence of the topologic 
features on the dynamics properties (i.e. attractors sepa- 
ration), we apply correlation analysis and path analysis. 
A series of interesting results are obtained regarding not 
only the separation indices, but also the importance of 
several local and global topologic features on the overall 
attractors separation. Although the current work con- 
centrates on attractors in grandmother dynamic systems, 
several of the results, measurements and methods can be 
immediately extended to other relevant systems based on 
complex networks, including those involving information 
transfer and retrieval. 

This article starts by presenting a brief review of 
more closely related works, and follows by describing 
the adopted notation and methodologies. Next, each of 
the four considered theoretic network models are briefly 
reviewed and their generation described. The diffu- 
sion procedure and 'differentiation' of the state values 
adopted to implement the attraction basins are described 
next, as well as the diffusion-base mechanism for activa- 
tion/retrieval of prototypes. The two separation indices 



are motivated and mathematically defined. The simula- 
tion results and respective discussion are then presented, 
followed by the correlation and path analyses of the ef- 
fects of the topologic features of the network on the re- 
spective attractors separability. Several interesting find- 
ings and effects are identified and discussed. The article 
concludes with a general synthesis of the reported inves- 
tigation and with identification of possible future devel- 
opments. 



II. A BRIEF REVIEW OF PREVIOUS 
DEVELOPMENTS 

The subject of attractor characterization has been ex- 
tensively addressed in the literature, so we limit ourselves 
to reviewing a small subset of those works which are more 
general or more directly related to the investigations be- 
ing considered in the present work. 

A comprehensive study of attractor networks has been 
presented by Amit [37]] • Torres et al. have consid- 
ered the storage capacity of attractor neural networks 
in which the synapses can undergo depression [38| and 
found that the memory capacity decreases with the in- 
tensity of the depression. Amit and Brunei [36| investi- 
gated learning in attractor networks considering the ef- 
fects of stimuli in a network with wired-up patterns. To- 
pographic neuronal networks involving grandmother cells 
have been extensively studied by T. Kohonen and collab- 
orators (e.g. [1^, HO, [4l[ ) in the form of the self-organizing 
map (SOM) or Kohonen networks. Here, the neurons are 
spatially distributed and develop specificity to prototype 
pattern stimuli by influencing the weights of neighboring 
nodes. Therefore, spatially adjacent smooth attraction 
basins are defined. 

Several works have brought together the two impor- 
tant areas of complex networks and dynamic systems, 
including neuronal networks. At least two excellent sur- 
veys have been written about this important issue 0, [s| • 
The problem of neuronal structure and dynamics has also 
been a subject of growing interest, including but by no 
means limited to g3, HI H |H, |H, E3, (3 EH, US- 
Stauffer et al. [5l[ investigated the performance of di- 
luted Hopfield networks underlined by the BA model. 
The influence of the network topology on the dynamics 
of neural networks has been investigated also by Costa 
and Stauffer [52| , who considered spatial neural networks 
and concluded that its performance increased with the 
spatial uniformity of the cells distribution. Torres et 
al. [53J , showed that the pattern capacity of an attractor 
neural network with scale free topology is higher than 
for a random-diluted network with the same number 
of connections. They also found that, at zero temper- 
ature, the performance of scale free nets improves for 
larger values of the power-law exponent. Models of non- 
randomly diluted neuronal networks whose connectivity 
is determined as a function of the shape of its individ- 
ual neurons (as well as their relative spatial positions) 
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has been reported by Costa and collaborators [3 l54j . 
It was found that the shape of individual neurons can 
have a great influence on the respective memory capac- 
ity, suggesting that shapes more similar to real neurons 
tend to have better attractor properties. The influence 
of the network topology on the recovery of patterns in 
recurrent neuronal networks was addressed by Castillo 
et al. [551] . Their work showed that the retrieval prop- 
erties can be enhanced by considering connectivity more 
structured than in random networks. Morelli et al. [5r| 
investigated the memory capacity in associative networks 
and found that the best performance is obtained at an 
intermediate level of disorder. Zhou and Lipowski (57j 
investigated, through analytic and simulation means, a 
general class of dynamic systems on scale free networks 
with binary states. They reported important variations 
of performance with respect to the scale free exponen- 
tial coefficient. The effect of structured connections on 
the interactive statistical mechanics algorithm for min- 
imization of the Bethe free energy (associated to Ising 
models) has been studied by Ohkubo et al. 58]. The 
adaptation of the Sznajd dynamics to take place over 
the network connectivity instead of its states has been re- 
ported by Costa [59j ] , yielding network realizations which 
are a consequence of an inherent dynamical process. Lu 
et al. [6(| considered the effect of regular, random, small- 
world and scale free topologies on Hopfield networks. 
They reported, among other findings, that the perfor- 
mance improved with the local order of the connections, 
which seems to be in agreement with 52] . The periodic- 
ity of activity in networks with small- world and scale-free 
topologies were investigated by Paula et al. [6l| who con- 
cluded, among other findings, that periodic activity ap- 
pears only for relatively small networks. Perotti et al. [62j 
studied the interesting problem of associative memory on 
a growing diluted Hopfield model which converges to a 
small- world, scale free topology and showed that the per- 
formance of such a network is higher than that of a ran- 
domly diluted network with the same connectivity. More 
recently, Davey et al. [o3j investigated sparse small world 
associative memory considering Perceptron training un- 
der small world connectivity ranging from local to global 
and found that non-symmetric connectivity networks ex- 
hibited superior performance. By considering random 
walks as a reference model for implementing dynamics in 
complex networks, Costa et al. investigated the correla- 
tions between the topology (node degree) and activation 
(frequency of visits to nodes at equilibrium) . They found 
that while full correlation is guaranteed for undirected 
networks, it can vary substantially in directed networks 
such as biological neuronal networks and the Internet, 
implying that topologic hubs are not necessarily hubs of 
activity. That work also identified a relationship between 
scale free networks and the Zipf's law [64j. A study of 
self-organizing models underlined by complex networks 
related to mental processes has been reported by Wede- 
mann et al. [65j . The investigation of the relationship 
between topology and dynamics at higher spatial scales 



(e.g. cortical areas) has also been addressed in an increas- 
ing number of works (e.g. [H M, H M, H [lO, Izl I?! ) • 

Increasing attention has also been drawn on the syn- 
chronization of oscillations as the means for pattern en- 
coding and retrieval (e.g. [74J). Among the works related 
to the separation of attractors, Arecchi [75[ has consid- 
ered a metric structure for the percept space, while taking 
into account the separation between states. Wang et al. 
investigated the influence of the node degree distribution 
in the synchronization of two-layer neural networks. The 
criticality of coupling parameters on the synchronization 
of an ensemble of identical neural networks with small- 
world topology has been addressed by Wang et al. [76| . 

All in all, as far as the influence of connectivity on 
the performance of dynamic systems for storing patterns 
is concerned, several of the above reviewed works seem 
to indicate that better results tend to be obtained by 
considering non-random connectivity, but at a level of 
order that ranges from low to intermediate. 



III. NOTATION, BASIC CONCEPTS AND 
METHODOLOGY 

This section covers the concepts and methods used in 
the current investigation. Its subsections present the ba- 
sic concepts and measurements in complex networks, the 
four considered theoretic network models, the procedure 
suggested to extend the prototype influence through their 
successive neighborhoods in order to establish the attrac- 
tion basins, the diffusive way to activate the attractors, 
the separation measurements, and the correlation and 
path analyses considered in this work. 



A. Complex Networks Concepts and Topologic 
Measurements 

A graph T involves a set V of N nodes interconnected 
by a set U of E edges, i.e. V = (V,U). Each directed 
edge linking a node i to a node j is represented as (i, j). 
Such a graph can be conveniently expressed in terms of 
its adjacency matrix K, so that K(j,i) = 1 whenever an 
edge exists (otherwise, K(j,i) — 0). A graph such 
that K(i,j) = K(j,i) = 1 is said to be non-oriented. Al- 
though the present work focuses on this type of graph, 
all reported developments can be extended to directed 
graphs. The graphs considered henceforth are also de- 
void of self-connections (i.e. K(i,i) = for every node 
i). A complex network is henceforth understood as a 
graph exhibiting a particularly intricated structure, in 
the sense of differing from a random graph of the Erdos- 
Renyi type [77], [78|. However, because of statistic fluctu- 
ations, a random graph can also exhibit intricated struc- 
ture. 

Given a generic node i, some measurements can be 
associated to it (e.g. [1, HQ)- Its degree k(i) is defined 
as the number of edges attached to it, so that 
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k{i) =Y,K{p,i). 



(2) 



The immediate neighbors of a node i correspond to 
those nodes which are directly attached to it. The clus- 
tering coefficient of a node i expresses the degree of con- 
nectivity among its immediate neighbors and can be cal- 
culated as 



cc{i) 



e(i) 



(3) 



where e(i) is the number of undirected edges between 
the immediate neighbors of i and e max is the maximum 
possible number of such connections, given as e max — 
e(i)(e(i) - l)/2. 

The shortest path between any two nodes i and j cor- 
responds to the minimal set SP of adjacent edges con- 
necting those two nodes. A network in which all nodes 
can be reached, through paths, from any other node is 
henceforth said to be connected. The respective shortest 
path length sp is defined as the number of edges in SP. 
Therefore, the immediate neighbors of a node i can be 
alternatively defined as the nodes which are at shortest 
path length of 1 from node i. The second neighborhood 
(or neighbors of second hierarchy) are those nodes which 
are at shortest path length of 2 from i, and so on. The 
maximum shortest path length between any pair of nodes 
is defined as the network diameter, henceforth abbrevi- 
ated as diam. 

Given a set R of M reference (or seed) nodes, it ispos- 
sible to obtain the respective Voronoi tessellation [79( of 
the network [35j with respect to the seeds, which parti- 
tions the N nodes into M connected subgraphs associ- 
ated to each of the reference nodes. This can be achieved 
by, for each node i, identifying which of the reference 
nodes is closest (in the sense of shortest path) to i and 
assigning it to that region. Given two nodes i and j, 
whose respective immediate neighbors are represented by 
the sets n(i) and n(J), we define their common neighbors 
as the set c(i) — n(i) Hn(j). Given the Voronoi partition 
of a network, it is interesting to devise a measurement ca- 
pable of expressing the uniformity of the areas A(i) (i.e. 
number of nodes) of each of the M partitioned regions 
i. This can be conveniently achieved by considering the 
geometric average [A] of the areas, i.e. 



M 



l/M 



[A] 



(4) 



\i=l 



Observe that the geometric average implies a high 
penalty on higher variations of the Voronoi areas, there- 
fore providing a more strict quantification of the homo- 
geneity of those areas. Figure [2] illustrates the compari- 
son between the arithmetic and geometric average consid- 
ering two measurements p (with < p < 1) and q = 1—p. 



The ability of the geometric average in quantifying the 
similarity between p and q is evident from this figure. 
While dispersion-related measurements (e.g. standard 
deviation and entropy of the state activations) could be 
used, they could not be interepreted as quantifications of 
the activations. 



leverages 
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FIG. 2: The arithmetic (*) and geometric (+) averages be- 
tween the values p and q = 1—p. The geometric average 
allows the quantification of the uniformity between the val- 
ues of p and q, with a peak at p — q — 0.5. 



As the geometric average u for the M Voronoi areas 
varies as < u < l/M, it is convenient to redefined [A] 
as 



l/M 



[A]=M[i[A(i) 



(5) 



i=l 



The matching index [66l. |67| of a pair of nodes i and j, 
expressing the relative degree of overlap between the im- 
mediate neighborhoods of those nodes, can be calculated 
as 



mi(i,j) = 



n(i) fl n(j) 
n(i) U n(j) 



(6) 



This measurement gives the fraction of immediate 
neighbors of i and j which are common neighbors to them 
both. For instance, for the situation depicted in FigurcUJ 
we have that the matching index of the pair of nodes PI 
and P2 is given as m(Pl, P2) = 2/8 = 0.25. 



B. Complex Networks Theoretic Models 

Four theoretic models of complex networks are consid- 
ered in the present work: Erdos-Renyi (ER), Barabasi- 
Albert (BA), Watts-Strogatz (WS) and a geographic 
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model (GG). Reflecting their different natures and orga- 
nizing principles, these models correspond to a significant 
portion of the complex networks found in nature, provid- 
ing therefore a representative choice of models for the cur- 
rent study of attractor separability. In order to account 
for a more coherent comparison between the separability 
of attractors in these four models, each comparison al- 
ways consid er N and (k) for each model to be as similar 
as possible jl02| |. In this work, the average degree of the 
BA model (defined by the parameter to) is always taken 
as a reference for defining the average degree of the other 
models. The methodology for constructing such models 
are described in the following subsections. 

1. Erdos-Renyi 

Random networks, also called Erdos-Renyi - ER, were 
among the first models of stochastic networks to be ex- 
tensively studied(e.g. [77], [78|, H3|). These networks are 
characterized by constant probability of having a connec- 
tion between any of the possible pairs of nodes, and are 
therefore related to Poisson processes. The connectivity 
of ER networks can generally be well approximated in 
terms of its average degree, implying that such networks 
are similar to regular networks, characterized by having 
the same degree at any of its nodes. As a consequence 
of its indiscriminate connectivity and largely regular or- 
ganization, the ER type of networks does not typically 
provide a good model of natural structures and phenom- 
ena, where the connections tend to follow more purposive 
and specific rules. 

In an ER graph, each possible connection between each 
possible pair of nodes, has constant probability 7 of ex- 
istence. Such networks can be easily created through 
Monte Carlo simulation by making K(i,j) = K(j, i) = 1, 
with i 7^ j, with probability 7. Observe that the con- 
struction of ER networks consider only two parameters: 
N and 7. In order to obtain values of (k) similar to the 
BA reference, we enforce 7 = 2m/ '(N — 1), where to is a 
parameter of the BA model (see next subsection). 

2. Barabdsi- Albert 

The so-called Barabdsi- Albert model 0, HH, HI " BA, 
belongs to the important class of scale free networks. 
This type of network is characterized by the fact that 
the loglog plot of their node degrees tends to a straight 
line, implying the absence of any characteristic scale. In 
other words, the node distribution in such a network fol- 
lows a power law. One of the most important properties 
of scale free networks (e.g. [3, |82j), when compared to 
models such as the ER, is the higher probability of ex- 
istence of hubs, i.e. nodes with particularly high degree. 
Such special nodes are particularly important in defining 
the connectivity and topologic features of the network, 
such as the average shortest path length. For instance, 



the fact that a hub connects to many nodes immediately 
implies that the shortest path between any of these nodes 
will be at most equal to 2 edges. Several important natu- 
ral and human-made structures - including the Internet, 
WWW, protein interaction and even scientific collabora- 
tions - have been found to exhibit the scale free prop- 
erty (e.g. [3, 0, UK])- The BA model incorporates the 
so-called rich-get-richer paradigm because of its attach- 
ment of links being preferential to the degree of existing 
nodes. 

In the current work, BA networks are generated start- 
ing with toO randomly connected nodes. At each subse- 
quent step, a new node with to edges is added to the net- 
work, with each of the to edges being attached to previous 
network nodes preferentially to their degree. Therefore, 
each new connection is more likely to be established with 
previous nodes with high degree, implementing the 'rich 
get richer' paradigm. As with the ER model, the con- 
struction of BA networks also involves only two parame- 
ters: N and m. 



3. Watts- Strogatz 

Historically, the small world networks of Watts and 
Strogatz [5], y33[ followed the random networks of Erdos, 
Renyi and collaborators. Small world networks are ex- 
actly as implied by their name, i.e. the average shortest 
path length between their nodes tends to be small. At 
the same time, they also tend to be characterized by rela- 
tively high clustering coefficient, implying that they local 
connectivity is relatively high. The small world property, 
which has been found to be present in many interesting 
networks including ER and BA, has important implica- 
tions for the separation of attractors because it implies 
that many prototype nodes will be near one another and, 
consequently, possibly less separated as far as the dy- 
namics is concerned. The WS model considered in this 
work, however, presents some specific topologic organiza- 
tion which has potential implications for the distribution 
of the prototypes. More specifically, this model is char- 
acterized by a relatively high regularity and uniformity 
of local connectivity. 

The Watts-Strogatz networks used in the current work 
have been constructed by starting with a ring of nodes 
where each node is connected to its to clockwise and 
anti-clockwise nodes. After such an initial network is 
obtained, a% of the existing connections are rewired at 
random. This network model involves three parameters: 
N, m and a. All configurations in this work assume 
a = 10%. 



4- A Geographic Model 

Geographic networks (e.g.[H H HI HI)) - 
also called spatial, geometric or topologic - are charac- 
terized by the fact that their nodes have well-defined spa- 
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tial positions within an embedding space. Frequently, the 
connectivity in such networks is considered to be highly 
influenced by the spatial adjacencies and/or spatial prox- 
imity between its nodes, in the sense that two nodes 
which are adjacent or near one another will have higher 
chances of being connected. Therefore, geographic mod- 
els in small dimensional spaces (e.g. 2D or 3D) tend not 
to be small world. Such a property is particularly im- 
portant as far as the separation of the prototype nodes is 
concerned because, in principle, this type of network al- 
lows more space distribution of nodes which are relatively 
further apart. Here, we consider one of the simplest pos- 
sible approaches to obtaining a geographic model, which 
involves the distribution of the N nodes uniformly along 
a 2D space followed by the interconnection of all nodes 
which are closer than a given distance d. 

In order to build a GG network, we start with an empty 
L x L discrete space S, such that each of its positions is 
expressed as S(x,y), where i and j are integer values so 
that 1 < x, y < N. This space is henceforth understood 
as a Poisson field with density p, in the sense that any re- 
gion with area a (i.e. number of discrete elements (a;, y)) 
will have, in the average, a total of ap points marked 
as S(x,y) = 1 The network nodes are selected 

by considering each position (x,y) in the space S with 
probability p = N/L 2 , implying an average total num- 
ber of nodes N. Then, each of such nodes, marked as 
S(x,y) — 1, is connected to all other nodes to be found 
up to a maximum Euclidean distance d. Therefore, the 
average degree of the network can be defined by control- 
ling d. More specifically, we make d = Ly / 2m/ (Ntt), so 
that every disk of radius r in S centered at each node will 
contain, in the average, (fe) = 2m, where m is the BA pa- 
rameter taken as the reference. The growth parameters 
of such a network model therefore are again limited to 
only N and m. In order to ensure a relatively small vari- 
ation of N, every generated network with a total number 
of nodes smaller than 90% of the desired value of N were 
discarded. Figure [3] illustrates a GG complex network 
obtained by using the methodology described above for 
N = 50 and m = 3. 



C. Generating the Attraction Basins 

As indicated in the Introduction, in order to avoid the 
intricacies and specificities of how each distinct dynamic 
system represents attractors, here we resort to a simple 
methodology involving diffusion of activity from the pro- 
totype nodes, followed by the transformation of the so 
obtained activity into a derivative network [H, [9(| ■ Al- 
though not reproducing in detail the attraction basins 
which would be otherwise produced by diverse specific 
dynamics, this approach does ensure the smoothness of 
coding, i.e. the property that nodes which are topologic 
close will tend to have similar state dynamics. The de- 
tails of such a methodology are presented as follows. 

Let the M prototype patterns be associated with re- 



spective nodes chosen with uniform probability among 
the N network nodes. Let P be the vector such that 
P(i) — 1 if and only % is one of the prototype nodes, 
with P(i) = otherwise. In order to allow a proba- 
bilistic interpretation, we normalize this vector as p — 
P j ^2^ =1 p{i)- This vector will act as the fixed source of 
probabilities during the diffusion. The adjacency matrix 
describing the network is also normalized into its respec- 
tive transfer matrix W, i.e.: 

W(i,j) = K(i,j)/k(i), (7) 

where k(i) is the degree of node i. Note that all the 
sums of W along each of its columns will now be equal to 
1, i.e. W is a stochastic matrix. In order to ensure that 
the matrix K is connected (irreducible) [lij , all the nodes 
which do not belong to the main connected component 
are excluded from the network at the end of its respec- 
tive construction. Because the considered average node 
degrees are relatively high, and well above the percola- 
tion critic density, very few nodes are removed through 
such a procedure. 

Now, the final distribution of occupancy ft T of each 
node i after T interactions can be calculated by applying 
recursively (T times) the following set of equations: 

8 = (ft+p) , 

= Wb. (8) 

The number T t of total interactions is henceforth de- 
fined as corresponding to 3 times the diameter of the 
respective network, i.e. T t — 3diam. Figure EJb) illus- 
trates the occupancy states obtained for a GG network 
with iV = 100, (k) = 6 and M = 2 after T t = 27 interac- 
tions. 

After the occupancy state of each node is obtained for 
the network, its respective derivative network A (e.g. (89l . 
l90j |) is obtained by applying the equation below for each 
edge (i,j) existing in the original network. 

aux — — 

aux > => A(j, i) = aux 
aux < => A(j, i) — aux/10 
aux = => A(j, i) = — 1 

The following substitution is applied afterwards: 

A{j, i) = -1 => A(j, i) = max(A)/p. (9) 

All cases in this work assumes (3 — 1000, but this pa- 
rameters has been found not to be critical. 
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Note that the values A(j, i) correspond to the weights 
of the respective edges Those edges which connect 

a node with small induced activity to a node with higher 
activity will have larger weights. More specifically, in 
case aux > the weights are directly proportional to the 
difference of activations. Edges leading from higher to 
smaller activations have one tenth of the reciprocal edge. 
Because several adjacent nodes may result with equal 
activations, the eventual previous connectivity between 
then will be preserved though at an incremental value 
proportional to the maximum activation (Equation [5]). 
Observe that the matrix A is not symmetric. 



D. Activating the Network 

Once the derivative network defined by the weight ma- 
trix A has been calculated, its activation can be eas- 
ily achieved through a random walk preferential to the 
weights of the edges (see also [9l| , where information is 
transmitted considering the gradient of a node). In or- 
der to do so, we obtain the stochastic version of A by 
applying the following equation to each of its edge 



S(j,i)=A(j,i)/^A(j,i). 
The activation after T steps is now given as 



a= l/N5 T l, 



(10) 



(11) 



where 1 is the N x 1 vector of ones. As before, we 
assume that the total number of interactions T r is equal 
to 3 times the network diameter. Therefore, the final ac- 
tivation corresponds to the near equilibrium occupancy 
of each node after starting from any node. Figure [31(c) 
shows the activations obtained by the method described 
above with respect to the network in Figure [2Kb). It is 
interesting to note that the peaks of the obtained ac- 
tivations do not necessarily correspond to the original 
prototype nodes. This is an interesting consequence of 
the fact that the activation of the derivative network is 
affected not onl y by the attraction basins, but also by 
the in-strength [1031 ] and the local connectivity of each 
node. More specifically in case the in-strength is 
equal to the out-strength for all nodes, the equilibrium 
activation will be directly proportional to the respective 
in-strength. However, this result is not guaranteed for 
asymmetric connectivity such as in the derivative net- 
work. 



E. Separation Indices 

Given a specific grandmother dynamic system, it is 
important to quantify in an objective manner the sepa- 



rability of its attractors. Recall that the system incorpo- 
rates M prototype patterns, associated to respective pro- 
totype nodes p — 1,2,..., M. In this work we propose 
two separation indices, Si n d and s ng h, defined by taking 
into account diffusion dynamics along the complex net- 
work underlying the dynamic system. For simplicity's 
sake, the latter is henceforth called simply as separation 
index and abbreviated as s. 

Once the activations have been obtained in each of 
the dynamic states associated to the nodes, we define 
the individual separation index Si n d as being proportional 
to the geometric average of the activations v(j>) at each 
prototype node p — 1,2,..., M, i.e. 



l/M 



Sind 



M 



l[v(i) 



(12) 



where the proportionality factor M is introduced in 
order to ensure that < w < 1 instead of < w < 
l/M. As with the quantification of the uniformity of the 
Voronoi areas (Section IIII A[) . the geometric average will 
reach its maximum when all activations v(i) have the 
same value. 

However, as preliminary simulations (see Section lTVBp 
showed that this index tends to be too small, an alter- 
native separation index has been considered which takes 
into account also the immediate neighborhoods of each 
prototype node. Consider the situation illustrated in Fig- 
ured] This figure shows two prototype nodes PI and P2 
as well as their common C immediate neighboring nodes. 
Because these nodes are at the same shortest path dis- 
tance from PI and P2, they do not contribute to the 
discrimination between those prototype nodes and are 
therefore not considered in the calculation of the neigh- 
borhood separation index, which is more formally defined 
as follows. 

Let Z(i) be the set including the respective prototype 
node i and its immediated neighbors which are not com- 
mon to the immediated neighborhoods of any of the other 
prototype nodes. The probability p{i) at this set of nodes 
corresponds to the sum of the normalized activations of 
the states in Z(i). Ideally, all prototype nodes should re- 
sult with probability equal to l/M, implying that all pro- 
totypes are equally accessible and therefore maximally 
separable. In order to quantify how the obtained network 
state approaches such a reference separation, we define 
the neighborhood separation index s ng h as correspond- 
ing to the geometric average of the total probabilities at 
each set Z(i), for all M prototype nodes i, i.e. 



l/M 



= M 



U.p(i) 



(13) 



where the multiplying constant M is, as before, in- 
cluded in order to imply that < s < 1. The maxi- 
mum separation between attractors is therefore obtained 
whenever s = 1. 
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(c) (d) 

FIG. 3: Example of GG network obtained for TV = 50 and m = 3 (with (k) = 5.5) (a). The respective occupancy states, 
defining the basis of the prototype nodes (striped), after 27 interactions (b). Note the smooth distribution of states through 
both the topologic neighborhoods. The activation induced by the random walks preferential to the weights of the respective 
derivative network (c). The Voronoi tessellation defined by the two prototype nodes (represented as striped) (d). 



F. Correlation Analysis 



Given several measurements of the topology of the net- 
work under analysis, as well as the separation indices, a 
first insight about their possible relationship and redun- 
dancy can be obtained by considering the Pearson cor- 
relation coefficient. Let us express each measurement as 
a random variable X of which we have Nx observations. 
First, we standardize [92| the variable X as 



X 



X 



a(X) 



(14) 



where [ix and cr(X) are the estimate of the average 
and standard deviation of X. The new, normalized vari- 
able has average zero and unit variance. The Pearson 
correlation coefficient r(X,Y) between any pair of nor- 
malized measurements X and Y can now be estimated 
as 
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FIG. 4: A portion of a hypothetical network showing two pro- 
totype nodes (PI and P2). The nodes belonging to both im- 
mediate neighborhoods are identified by the region C, while 
the nodes belonging to the respective immediate neighbor- 
hoods minus the common nodes are enclosed by the regions 
Nl and N2, respectively. 



case of the SEM framework, in order to obtain indica- 
tion about the influence of the several measurements of 
the topology of the networks on the respective attractor 
separation indices. 

Path analysis was largely developed by S. Wright 
(e.g. [97} ) in order to model explanatory relationships 
between observed variables. This methodology is similar 
to the solution of a system of equations implied by sub- 
stituting the model generated covariance matrix into the 
sample covariance |95| . One of the interesting features of 
this approach is that it considers the influence of the co- 
variances between all variables, sometimes being closely 
related to multivariate linear regression. The path anal- 
ysis performed in this work considers the structural rela- 
tionship between the topologic and dynamic properties of 
the investigated networks as shown in Figure [5] A rela- 
tively simple relationship between the measurements has 
been considered, where the topologic features are under- 
stood to cause the dynamic properties, namely the in- 
dividual and neighborhood attractor separation indices 
(sind and s, respectively). Each of the measurements are 
associated to a reference number (upper righthand side 
of each box). The parameters jij, which are in princi- 
ple not known, express the importance of the topologic 
measurements with respect to the separation indices. 



N 



r(X, Y) 



N x 



(15) 



Observe that -1 < r(X, Y) < 1, while r(X,Y) = 
means lack of correlation between the two measurements. 
It is important to stress that uncorrelation does not im- 
ply statistic independence. At the same time, correlation 
can by no means be understood as a certain indication 
of causality. In case two measurements result with high 
absolute value of the Pearson correlation coefficient, they 
are said to be correlated, indicating redundancy of mea- 
surements. However, even correlated measurements can 
contribute to the characterization and discrimination be- 
tween the networks 91. 



G. Path Analysis 

While the Pearson correlation coefficient quantifies, in 
a normalized fashion, the joint variation of two measure- 
ments, such a pairwise measurement does not consider 
information about additional measurements. A series of 
sound statistic methods, ranging from multivariate re- 
gression (e.g. [93L l94j]) to the more sophisticated Struc- 
tural Equation Modeling (SEM) (e.g. [95l [96|), can be 
considered in order to obtain a more representative char- 
acterization of the relationship between multiple random 
variables (including latent variables) or measurements. 
In this work we considered the Path Analysis method- 
ology (e.g. [95|, [9I, [97]]), understood here as a particular 
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FIG. 5: The structural relationship between the topologic 
and dynamic measurements considered for the path analysis 
reported in this work. The covariances between variables 3 to 
9 are not shown for simplicity's sake. 



Equations [TB] and [T7] express the relationship between 
the considered variables, reflecting the fact that the re- 
gression coefficients 7,j establish the weights of the influ- 
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ences of each topologic variable on to th e two separation 
indices. The environment LISREL [l04j was used in this 
work in order to perform path analysis. 

Sind oc 713 (cc) + 714 (sp) + ji 5 osp + Ji 6 Vi 

+717 (mi) + 7i 8 cr(mi) + j 19 diam (16) 

s oc 713 (cc) + 714 (sp) + Jucrsp + ji e Vi 

+7i 7 (mi) + jis<r{mi) + j 19 diam (17) 

IV. RESULTS AND DISCUSSION 

This section starts by identifying each involved param- 
eter which can affect the simulations and follows by pre- 
senting the results obtained for specific configurations of 
N and m and an analysis of the variation of the param- 
eters. The statistic analysis of the relationship between 
the topologic and dynamic measurements by using cor- 
relation and path analyses is also reported. 



A. Involved Parameters and Their Expected 
Effects 

An important first issue to be considered while inves- 
tigating the attractors separation in complex dynamic 
networks concerns the identification of all involved pa- 
rameters which can influence the results. The parame- 
ters involved in our simulations, as well as their expected 
effects, are described in the following: 

The network model and its intrinsic parameters: Each 
theoretic model of complex network considered in this 
work is likely to yield different attractors separations. 
Those networks characterized by the small world prop- 
erty (i.e. ER, BA and WS) are, in principle, likely to 
produce less separated attraction basis because of the 
relatively small shortest path distance, expected in the 
average, between the prototype nodes (however, see Sec- 
tion Ell)- While the ER, BA and GG models can be 
generated by considering just two parameters (i.e. N and 
m), the WS network also involves a third parameters cor- 
responding to the percentage of rewirings, which is fixed 
as a = 10%. 

The network size N : This is an important global pa- 
rameter of every complex network, corresponding to its 
total number of nodes. One of the most important as- 
pects related to this parameter are the so-called finite 
size effects, namely the fact that the network properties 
change considerably when moving from large (possibly 
infinite) to small values of N . As an extreme exam- 
ple, the connectivity of any network will decrease when 
N approaches just one or two nodes. Although in this 
work we are more interested in finite size networks, it is 
often useful to try to extrapolate from properties mea- 
sured for smaller values of N to the infinite limit, as 



done in Section TlV CI Because of the computational de- 
mand required to simulate hundreds of realizations for 
each configuration, the current work is limited to rela- 
tively small values of N. As far as the attractors separa- 
tion is concerned, it is expected that, for fixed m and M, 
the separability will tend to increase with N because of 
the additional space thus allowed for the representation 
and distribution of the prototypes. 

The network average node degree (k): This parameter, 
which in this work is defined with respect to the refer- 
ence m, is particularly important in defining the overall 
degree of connectivity in the network, especially in those 
structures which are not scale free and therefore have spe- 
cific degree scales. Generally, smaller values of (k) tend 
to imply longer shortest paths, possibly improving the 
attractors separation (however, see Section TlV E[) . This 
parameter is not systematically investigated in this work, 
which mostly considers fixed m = 3 (however, a situation 
with m = 10 is considered in Section TlV Bp . 

The number M of prototype patterns to be represented: 
This parameter is directly involved in the separability of 
the prototype patterns, in the sense that the larger the 
number of prototypes, the less separated they tend to be. 

The number T t of interactions used for diffusion 
around each prototype node: The diffusion of activity 
emanating from the prototype nodes has been verified 
to converge quickly for all considered networks, so that 
the assumption of the total number of interactions to be 
given as T t = Zdiam practically ensures the resulting ac- 
tivity to correspond to its equilibrium state. 

The number T r of interactions considered in the re- 
trieval random walk: Again, the assumed total number 
of interactions used in the activation of the attraction 
basins is large enough to imply near equilibrium states. 



B. Fixed Configurations 

In order to get a better understanding about the sep- 
arability of the attractors in the considered four theo- 
retic models and to obtain preliminary indications about 
the effects of the involved parameters, we considered a 
set of preliminary simulations as described in the fol- 
lowing. All results presented in this section considered 
Tt = T r = Sdiam and 500 realizations of each configura- 
tion. 

So as to have the first glimpses of the separability in 
each of the four complex networks models, we considered 
N = 100, (k) = 3, and a relatively small number of proto- 
types M = 3. The individual separation indices Si n d and 
s obtained for each of the four models are shown in terms 
of their respective population histograms in Figures [5] 
and [7] The respective averages and standard deviations 
are also shown inside each graph. The first important 
result is that the individual separation index Si„d allows 
considerably less resolution than the neighborhood index 
s, with most of its values being close to zero (Fig. 0. 
As this effect has been confirmed for other configura- 
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tions, we henceforth focus attention on the neighborhood 
separation index s. Interestingly, similar distributions 
have been obtained for the pairs of models ER/BA and 
WS/GG in Figure [3 the former being characterized by 
smaller averages of s. Observe also the presence of cases 
where s = in the WS and GG cases. The largest av- 
erage, implying the better average attractor separability, 
was obtained for the WS model, followed by the GG net- 
works, which also implied a significant number of null 
separability indices. This phenomenon is caused when- 
ever all immediate neighbors of the prototype nodes are 
common. These results also show that a reasonable sepa- 
ration between the three prototype attractors have been 
obtained for the WS and GG cases. 

Next, we verify the possible influence of the network 
size N by considering the same configuration as before, 
but with N — 200. The respective results are shown in 
Figure [H It is clear from these results that the larger 
size of the network had relatively little influence on the 
separation indices, suggesting the finite size effect to be 
small for this number of parameters. 

Now we turn our attention to the influence of the num- 
ber of prototype patterns to be represented. The same 
configuration as in Figure [7] is considered, except that 
M = 10. The obtained results are shown in Figure [Sj It 
is clear that increase by more than threefold of the num- 
ber of prototype patterns implied not only a substantially 
more cases such that s = 0, but also the distributions to 
be strongly left-shifted in comparison with the respective 
cases in Figure [7] This effect holds for all the four con- 
sidered network models. The main reason behind such a 
substantial decrease of performance is that simply there 
was not much space left, in the average, between the pro- 
totype attractors. 

Because the last configuration exhibited an accentu- 
ated loss of separability because of lack of space in the 
network, it is interesting to reconsider the effect of in- 
creasing N for this situation. The results obtained for 
the same previous configuration, but now with N = 200, 
are shown in Figure 1101 The separation index increased 
substantially for all the four network models. It could 
be expected that such improvements tend to decrease for 
still larger N, until reaching a regime where little im- 
provement is observed. In such a state, the prototype 
nodes would be sufficiently far away one another so that 
their separation no longer depends on N. Additional in- 
vestigation about the change of the average and standard 
deviation of s are to be found in Section II V CI 

Finally, we check for the possible effect of the average 
node degree on the separation index. Again, the same 
configuration as the network in Figure [7] is adopted, but 
now with (k) — 20 (i.e. m = 10) instead of (k) = 6. Fig- 
ure [11] depicts the respectively obtained results, which 
indicate a clear reduction of the attractors separability. 
Such an effect is possibly a consequence of the fact that 
once the network become too intensely connected, the 
shortest path between the prototype nodes will be re- 
duced and the separability undermined. 



C. Finite Size Effects 

The simulations discussed in the previous section seem 
to have indicated that increases of the value of N would 
tend to improve the attractors separation until reach- 
ing a regime where so much space is available that the 
patterns no longer feels the finite size of the network. 
In order to obtain further insights about this effect, the 
configuration involving m = 3 and M — 3 was simulated 
for N = 50, 100, 150, 200, 250 and the results are given in 
Figure [12J This figure shows the average (a) and stan- 
dard deviation (b) of the neighborhood separation index 
in terms of N. It is clear from these results that the 
increase of N does enhance the attractors separation un- 
til reaching a plateau, where a possible 'unsaturation' 
occurs, indicating that the finite size effects are over for 
the specific parameters m and M. At the same time, a(s) 
decreases, also reaching a relatively low plateau. Simi- 
lar results could be expected for other configurations. In 
the case of a larger value of M, it is expected that (k) 
will increase more steeply along the smaller values of N, 
reaching a similar plateau at larger values. Therefore, as 
the 'unsaturation' effect was further corroborated by the 
additional analysis, such a behavior can be considered as 
a guideline for choosing a proper value of N given M and 
m, for instance by choosing N where (k) reaches a fixed 
proportion of its plateau value. 



D. Correlation Analysis 

In addition to studying the effects of the involved topo- 
logic measurements on the attractors separation, it is 
also particularly useful to try to identify in which ways 
the separation index is related to them. This may allow 
the prediction of the separability without performing dy- 
namic simulation, i.e. by considering only the topologic 
measurements. Table U show the Pearson correlation co- 
efficients obtained considering all pair os topologic mea- 
surement (i.e. (cc), (sp), cr(sp), Vi, {mi), a (mi) and 
diam) and the two separation indices (si n d and s). 

As previously verified in [9(, the pattern of correla- 
tions resulted not similar for each model of network. In 
addition, the obtained results indicate several high abso- 
lute correlation values. In the case of ER networks, the 
most intense negative correlation (-0.79) was obtained 
between (sp) and (mi), which was indeed expected as 
longer shortest path lengths tend to reduce the match- 
ing index. While similar high correlation (-0.78) was ob- 
served also for BA, this specific correlation was relatively 
smaller for the WS and GG cases (-0.50 and -0.42, re- 
spectively). This fact was reflected in the intense neg- 
ative correlation (-0.77) between s and (mi) and posi- 
tive correlation (0.63) with (sp) for the ER model. A 
similar effect can be observed for the BA, WS and GG 
networks. Observe that (mi) is highly correlated (0.72 
for ER) with a(mi), suggesting a dependence between 
the standard deviation and average of this measurement. 
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The index quantifying the uniformity of the Voronoi tes- 
sellation was found to be negatively correlated with (mi) 
and a (mi), as expected, because higher mi values are fa- 
vored by more irregular Voronoi areas. Interestingly, low 
correlation values were observed between s and the net- 
work diameter and clustering coefficients in all cases. It 
is also interesting to note that rather different patterns 
of correlations were obtained between Sj n( j and s and the 
topologic measurements for all network models. 



E. Structural Equation Modeling (SEM) / Path 
Analysis 

Although the correlation analysis described in the pre- 
vious section can provide interesting information about 
the pairwise relationship between the separation indices 
and the topologic measurements, such results are lim- 
ited because they do not reflect the general relationship 
between all the topologic measurements and the sepa- 
ration indices. In addition, some of the high observed 
absolute correlation values can be a consequence of spu- 
rious [HI, [9(| effects between the variables. In order to 
gather additional insights about the how the dynamic pa- 
rameters (i.e. the separation indices) are influenced (and 
even to a large extent defined) by the topology of the net- 
work, while considering all measurement co- variations, 
we performed path analysis considering the structural 
dependence between measurements as expressed in Fig- 
ure El It is expected that, by considering all dispersions, 
the path analysis can provide a more objective and fil- 
tered indication of the influences of the topologic mea- 
surements on the attractors separation. 

The considered data were respective to ER, BA, WS 
and GG networks with N = 100, M = 3 and m = 3. 
After estimation of the covariance of the measurements, 
coding and execution in the LISREL environment, the 
regression coefficients and residues (El and E2), as well 
as the relationship between these residuals (C12) were 
obtained. The results are given in Table HI1 

A series of interesting insights have been derived from 
these results. As with the correlations, the dependencies 
between the topology and separation are strictly specific 
to each network model, a dependency which may also 
change for other configurations with different values of 
the parameters M and m. In the case of the ER networks, 
the standard deviation of the matching index (a(mi)) re- 
sulted particularly influent (negative influence = -0.96) 
on the s index. At the same time, the average match- 
ing index ((h)) was found to have a strong influence on 
s. Except for relatively smaller negative influence of the 
(cc) on s, no particularly strong influences are observed 
with respect to the other topologic measurements. Such 
a strong influence of the matching index can be under- 
stood because non-zero matching index are obtained only 
in extreme cases, where the prototype nodes are too close. 
Therefore, nonzero matching indices are a strong and se- 
cure indication of poorly separated attractors. For this 



reason, the matching index dominated the path analysis 
and implied smaller influences for most other measure- 
ments, including the shortest path. This effect can also 
be observed for the BA and WS cases. However, it is 
interesting to note the weak influence of a(mi) on s in 
the case of GG networks. For the BA case, the strongest 
influence on s was identified for the Voronoi index Vi, 
which is compatible with the fact that higher uniformity 
of the Voronoi tessellation by the prototype nodes tends 
to promote better separation between those nodes. In- 
terestingly, a particularly strong influence of Vi on the s 
has been observed only for the BA model. This effect can 
be related to the fact that the BA provides the poorest 
general separation between attractors as a possible con- 
sequence of the generalized connectivity implemented by 
the hubs. In such cases, where a larger number of non- 
zero matching indices are therefore obtained, the Voronoi 
separation may become more relevant as a predictor of 
the separability. In the case of WS models, strong influ- 
ence (positive = 0.65) was obtained for the (cc). This 
effect is particularly interesting because it could be ex- 
pected that, by promoting smaller shortest paths, this 
measurement would be inversely related to the separa- 
bility, which is indeed the case for the respective ER and 
BA path analysis results. Indeed, a very weak depen- 
dency with this variable had been revealed by the Pear- 
son correlations. However, the positive influence of (cc) 
on s can be a consequence of the fact that GG networks 
with higher average clustering coefficients will tend to 
have more intense local connectivity which could allow 
the activity to diffuse more uniformly and to concentrate 
effectively around the prototype nodes. Such an effect 
would be more definite in the WS case because of the 
higher uniformity of local connections implied by its ring 
structure. The influences of the topologic features on the 
s obtained for the GG cases are dominated by (mi) as 
discussed above. 

While the previous path analysis has revealed a series 
of insights about the influence of the network topology 
on the dynamic separation of its attraction basins, it was 
strongly biased by the critical influence of the match- 
ing index. In order to try to get further insights on the 
structure/dynamics relationship for the four considered 
models, we repeated the path analysis while not includ- 
ing (mi) and a(mi). The results are summarized in Ta- 
ble [TTTJ Interestingly, except for relatively small increases 
with respect to Vi and (cc), the obtained influences re- 
mained similar, with small effects observed for the short- 
est path measurements. This can be understood as a 
confirmation that the separation between the attractors 
is critically defined by the overlap between the hierar- 
chical neighbors of the prototype nodes, especially their 
immediate neighbors (implying higher matching indices). 
In the cases where the matching index is zero, the lack of 
congruence between the original attractors and the dis- 
tribution of diffusive activity should be largely explained 
by the higher influences observed in Table HTT1 which in- 
volve the average clustering coefficient of the network and 
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the Voronoi index for the prototype nodes. Indeed, in the 
cases where the prototype nodes are not topologically too 
close one another, the lack of agreement between the ac- 
tivity distribution and the original attractors (as in Fig- 
ure [TJd and c) will depend particularly on the properties 
of the local connectivity as expressed by the clustering 
coefficient and the node degree. 



V. CONCLUSIONS 

The investigation about the relationship between the 
structure and function of networks (e.g. 0,H, repre- 
sents one of the most interesting perspectives for obtain- 
ing insights about complex dynamic systems. As briefly 
reviewed in this article, several works have addressed the 
problem of how the structure of the connectivity may 
affect and largely define the properties of dynamic sys- 
tems. One particularly important aspect which has re- 
ceived relatively lesser attention concerns the separabil- 
ity between different grandmother attractors, each rep- 
resenting a prototype pattern or state. This issue is crit- 
ically relevant because it has great impact on the capac- 
ity of the network for proper representation of patterns, 
the level of redundancy/robustness of such representa- 
tions, the degree of generalization for recognition of not 
previously trained prototypes, as well as the effective- 
ness during retrieval and activation of such prototypes. 
While other types of coding can be used in dynamic sys- 
tem, grandmother representation stands out as particu- 
larly important because it seems to be the way a great 
part of the primates cortex is organized. 

The current work has reported an approach to the 
characterization of the separability between prototype 
patterns which incorporates a number of special features. 
First, we have considered four representative theoretic 
models of complex networks - namely the random net- 
works of Erdo-Renyi, the scale free model of Barabasi- 
Albert, the small-world networks of Watts-Strogatz, as 
well as a simple, non-small-world topographic model. 
Second, by using a generic diffusive process, followed 
by the calculation of the respective derivative network 
in order to obtain the attraction basins, we obtained 
a methodology which completely avoids the intricacies 
and specifities implied by each type of dynamic system. 
While it remains to be verified how good accurate and 
general such an approximation is, it does allow the defi- 
nition of smooth attraction basins around each prototype 
node in a way which is remindful of many important dy- 
namical systems such as Kohonen's self-organizing maps 
and the primates cortex. Once the attraction basins are 
so defined, a simple diffusive scheme emanating from each 
network node is employed in order to obtain the general 
activation of the network. Although ideally such a pro- 
cedure should activate equally only the original proto- 
type nodes, the structured connectivity of the networks 
will act so as to produce non-uniform activation, where 
just a fraction of the overall activation coincides with the 



prototype nodes. The disagreement between the original 
prototypes and the induced activity has been quantified 
in terms of two separation indices, namely the individual 
and neighborhood indices. Because the former consider 
only the agreement and balance between the induced ac- 
tivation and the original prototypes, it resulted to be too 
small and therefore with low resolution for quantifying 
the attractors separation. By considering also the imme- 
diate neighborhood of the prototype nodes, the second 
index allowed a more informative indication of the at- 
tractors separability. 

The specific topologic features of each of the four con- 
sidered theoretic network models can have different ef- 
fects for the separation of the attraction basins. There- 
fore, we performed an investigation of the effects of the 
most important parameters of the simulations over the 
respective performance. We verified that the separation 
tends to decrease with the number of prototypes and in- 
crease with the size of the network. At the same time, 
more intense general connectivity, as expressed by the 
average node degree, also tended to undermine the at- 
tractors separation. Special attention was given to the 
finite sizes implied by the parameter N. The obtained 
results seem to suggest that the separability tends to in- 
crease with N up to a regime where the finite size of 
the network is no longer felt ('unsaturation'). Therefore, 
by identifying the region where such a plateau of sepa- 
rability is reached, it is possible to obtain near-optimum 
values of TV for given M and m. Next, we applied cor- 
relation analysis in orde not only to identify the inter- 
relationships between each topologic measurement, but 
also between these and the two separation indices. The 
presence of such correlations can not only help to under- 
stand the origin of the attractors separation, but also al- 
low the prediction of such a property from measurements 
of the network topology, without the need of simulations. 
Several tendencies were identified through such an anal- 
ysis, including the tendency of the index s to be strongly 
proportional to the uniformity of the Voronoi areas, as 
expressed by the Voronoi index Vi. A positive, though 
weaker, correlation was also identified between s and the 
average shortest path length (sp) of the networks. A 
strong negative correlation was identified between s and 
the matching index mi. Although all such behaviors are 
compatible with what could be expected, meaningfully 
different correlations differences were observed for each 
network model. Interestingly, the network model allow- 
ing the best overall separation of attractors was found to 
be the Watts-Strogatz, followed by the geographic, ran- 
dom and scale-free models. It is conjectured here that 
the superior properties of the WS model stem from its 
enhanced uniformity and low randomness of local connec- 
tivity as well as by the fact that, although being a small 
world model, there are relatively very few long range con- 
nections between any pair of nodes. Similar properties, 
though at the expense of a weaker local order and uni- 
formity of local connections, are characteristic of the ge- 
ographic model, which came in second in performance. 
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As a matter of fact, the WS and GG models showed sim- 
ilar behavior as far as the histograms of separation in- 
dex were concerned. The same was observed for the ER 
and BA cases. Therefore, from the perspective of attrac- 
tors separation, the WS/GG and ER/BA models seem to 
represent two different classes of systems, with the latter 
providing rather poorer separability. Interestingly, the 
small world property can not explain such a partition, 
as the GG is not a small world model as the other three 
cases. Consequently, it seems that the common denom- 
inators in the pairs WS/GG and ER/BA seems to be 
more strongly related also to the uniformity of the local 
connectivity. Such an explanation would be largely in 
agreement with several of the results of works investigat- 
ing the effect of connectivity on memory, as reviewed in 
Section HJ 

In order to try to learn more about the effects of the 
topological features of the network on the dynamical 
property of attractors separation, path analysis was also 
applied assuming a simple structural relationship. To 
our best knowledge, this is the first time such an insight- 
ful analysis has been applied for the study of complex 
networks. One of the main advantages of such an ap- 
proach over the Pearson correlation coefficients is that 
here the dispersions of all the considered measurements 
are taken into account instead of the pairwise relation- 
ships underlying the correlations. Interestingly, the path 
analysis yielded influences of the topologic measurements 
on the separation indices which were often substantially 
different from those suggested by the Pearson correlation 
coefficients. Of particular interest was the near null in- 
fluence assigned to the average and standard deviation 
of the shortest path length between the prototype nodes. 
This is all the most surprising as it seems intuitively rea- 
sonable to discuss much of the effect of the topology of 
the network over the attractors separation by considering 
such measurements. More specifically, networks where 
the prototype nodes are topologically close one another 
would tend to have poorer separation. However, the per- 
formed path analysis substantially emphasized, for all the 
four considered models, the importance of the average 
matching index on the attractors separation. On second 
thoughts, this is indeed reasonable because the presence 
of non-zero matching index is a certain indication of over- 
lap of the attraction basins. Even when repeating the 
path analysis while leaving out the matching index mea- 
surements, the other parameters (especially the shortest 
path length) did not result with higher influences. Other 
interesting insights, including influences which were spe- 
cific to network models, were also allowed by the path 
analysis, corroborating therefore the potential value of 
such a statistical approach in order to get insights about 
the relationship between structure and dynamics of com- 
plex dynamic systems. However, it is important to keep 
in mind that every statistical methodology should be un- 
derstood as a source of insights to be further investigated 
and corroborated rather than spelling definitive facts. 

Despite the relative comprehensiveness of the presently 



reported investigation, and perhaps as its consequence, 
a series of future developments can be suggested. First, 
it would be interesting to extend the reported investi- 
gation to larger network sizes and to consider additional 
measurements such as the standard deviation of the clus- 
tering coefficient and the spectral structure of the ad- 
jacency and weight matrix. Second, the sources of the 
deviations of the induced activation from the prototype 
nodes could be further investigated by considering simu- 
lations involving a single attractor. In such cases, all the 
loss of 'separability' would necessarily be a consequence 
of the probability leakage from the prototype node into 
its neighbors. It would be particularly interesting to ver- 
ify, through path analysis, which of the topological mea- 
surements of the network would be more influence on the 
attractor activation. It would also be worth investiga ting 
the potential of using the percolation transform [98l . 1991 ] 
over the network as the means to explain and predict the 
attractors separability. Other promising possibilities in- 
clude the extension of the matching index to take into 
account hierarchical levels larger than one, i.e. including 
also the second and higher neighborhoods. 

Several real problems are closely related to the issue 
of attractor separation and prototype activation. One 
real problem which could be particularly interesting to 
be addressed is the phenomenon of facilitation of neu- 
rons (represented by nodes). By facilitation of a neuron 
it is meant that that neuron will become more likely to 
engage into activity. For instance, the definition of tem- 
porary priorities in the primates brain could be related to 
the reinforcement of one or more attractors, so that they 
become more likely to be revised along time (e.g. through 
a random walk). Interestingly, as suggested by the cur- 
rent work, the facilitation of a given node would be highly 
dependent on its local connectivity. Such studies could 
be eventually extended to higher level mental dynamics, 
such as those underlying attention and even pathologies 
(e.g. [65|]). It is also reasonable to expect that the def- 
inition of attractors is a process which co-evolves with 
the network topology. In this sense, it would be inter- 
esting to try to identify growing schemes where the con- 
nectivity is affected by the success of the establishment 
of prototype attractores, and vice- versa. Another related 
investigation would be the identification of topologic or- 
ganizations of networks allowing optimal or near-optimal 
attractors separation. 



List of Symbols 

r = a graph or complex network; 

TV = number of nodes in a network; 

K = the adjacency matrix of a complex network; 

k(i) = degree of a network node i; 

m = the parameter in the BA model defining its average 
node degree. This parameter is considered as reference 
for all network models in this work; 
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diam = the diameter of the network; 

cc(i) = clustering coefficient of a network node i; 

sp(i,j) shortest path between nodes i and j in a complex 
network; 

mi(i,j) = the matching index of nodes i and j; 

Vi = the Voronoi index of separation between the areas 
of influence of the prototype nodes; 

(a) = the arithmetic average of the property a; 

[a] = the geometric average of the property a; 

a(a) = the standard deviation of the random variable a; 

M = the number of prototype patterns to be represented 
in a network; 



a = the activations in a network after a long random 
walk; 

Sind = the separability index between prototype nodes in 
a network at the level of individual nodes; 

s = the separability index between prototype nodes in a 
network considering also the immediate neighborhood of 
such nodes; 
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FIG. 6: The population histograms of the separation index s ind for the four considered theoretic models of complex networks 
with N — 100, M = 3 and (k) = 6 (i.e. m = 3). It is clear from these histograms that the separation index s ind does not 
provide good resolution. 
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FIG. 7: The population histograms of the separation index s for the four considered theoretic models of complex networks with 
N = 100, M = 3 and (k) = 6. Similar results were obtained for the ER/BA and WS/GG models. The former case involves 
smaller average and standard deviation of the s index. 
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FIG. 8: The population histograms of the separation index s for the four considered theoretic models of complex networks with 
N — 200, M = 3 and (k) — 6. Similar results were obtained for the ER/BA and WS/GG models. The results are similar to 
those obtained for N = 100 (see Figure [7]). 
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FIG. 9: The population histograms of the separation index s for the four considered theoretic models of complex networks with 
N = 100, M = 3 and (k) = 6. A substantial decrease of the attractors separation has been implied in all cases. 
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FIG. 10: The population histograms of the separation index s for the four considered theoretic models of complex networks 
with N — 100, M = 10 and (k) = 6. A substantial decrease of the attractors separation has been implied in all cases. 
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FIG. 11: The population histograms of the separation index s for the four considered theoretic models of complex networks 
with N — 100, M = 10 and (k) — 20. The effect of increasing the average degree was to substantially decrease the attractors 
separation in all cases. 




FIG. 12: The average and standard deviation of the neighborhood separation index ({k} and a(s), respectively) as a function 
of N for the four models of networks considering m = 3 and M = 3. While the averages increase and then tend to a plateau, 
the standard deviations decrease and also stabilize. 
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TABLE I: Pearson correlation coefficients obtained for the ER (a), BA (b), WS (c) and GG (d) networks with N = 100, m = 3 
and M = 3 considering the individual (s in d) and neighborhood (s ng h) separability indices and seven measurements of the 
topology of the network (i.e. average clustering coefficient ((cc)), average shortest path between prototype nodes ((sh p ath)), 
standard deviation of the shortest paths between the prototype nodes (a(sh p ath))), average Voronoi index (Vi), average 
matching index between prototype nodes ((m)), standard deviation of the matching index between prototype nodes (a (mi)) 
and diameter of the networks (diam) 
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TABLE II: The influences of each considered topologic measurement on the attractors separation indices as revealed by path 
analysis considering N = 100, M — 3 and m — 3. 
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TABLE III: The influences of each considered topologic measurement on the attractors separation indices as revealed by path 
analysis considering N = 100, M — 3 and m = 3. 



